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Abstract 

Background: Health economic evaluations support the health care decision-making process by providing 
information on costs and consequences of health interventions. The quality of such studies is assessed by health 
economic evaluation (HEE) quality appraisal instruments. At present, there is no instrument for measuring and 
improving the quality of such HEE quality appraisal instruments. Therefore, the objectives of this study are to 
establish a framework for assessing the quality of HEE quality appraisal instruments to support and improve their 
quality, and to apply this framework to those HEE quality appraisal instruments which have been subject to more 
scrutiny than others, in order to test the framework and to demonstrate the shortcomings of existing HEE quality 
appraisal instruments. 

Methods: To develop the quality assessment framework for HEE quality appraisal instruments, the experiences of 
using appraisal tools for clinical guidelines are used. Based on a deductive iterative process, clinical guideline 
appraisal instruments identified through literature search are reviewed, consolidated, and adapted to produce the 
final quality assessment framework for HEE quality appraisal instruments. 

Results: The final quality assessment framework for HEE quality appraisal instruments consists of 36 items organized 
within 7 dimensions, each of which captures a specific domain of quality. Applying the quality assessment 
framework to four existing HEE quality appraisal instruments, it is found that these four quality appraisal instruments 
are of variable quality. 

Conclusions: The framework described in this study should be regarded as a starting point for appraising the 
quality of HEE quality appraisal instruments. This framework can be used by HEE quality appraisal instrument 
producers to support and improve the quality and acceptance of existing and future HEE quality appraisal 
instruments. By applying this framework, users of HEE quality appraisal instruments can become aware of 
methodological deficiencies inherent in existing HEE quality appraisal instruments. These shortcomings of existing 
HEE quality appraisal instruments are illustrated by the pilot test. 



Background 

The optimal allocation of scarce resources for the pro- 
duction of health benefits to society requires best evi- 
dence of cost-effectiveness, and is relevant to any 
decision in health care. Health economic evaluations 
support the health care decision-making process by 
providing information on costs and consequences of 
health interventions. For example, the NHS Economic 
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Evaluation Database currently contains over 11,000 qual- 
ity assessed economic evaluations, the results of which 
are increasingly used for pricing and reimbursement 
decisions. 

To be useful, health economic evaluation studies 
should be methodologically comparable, of high quality 
(e.g., in terms of transparency, quality of data sources, 
completeness of documentation), and relevant for the 
health care decision context. However, the quality of the 
conduct and reporting in such studies varies [1]. Health 
economic evaluation studies are heterogeneous with re- 
spect to purposes, conceptual and measurement issues, 
and value judgments leading to problems with compar- 
ability and suboptimal delivery of care. To ensure the 
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scientific quality of such studies, and to facilitate the 
comparison and transferability of economic evaluation 
results, methodological standards for health economic 
evaluations have been established [2]. Besides the pur- 
poses of setting methodological and ethical standards, 
such standards are also used as a formal requirement 
prior to reimbursement [3]. 

There are different instruments to guide the quality 
assessment of health economic evaluations. Among 
these health economic evaluation (HEE) quality appraisal 
instruments, considerable agreement exists on the ter- 
minology of economic evaluation, the importance of 
considering alternatives, the need for specifying the ana- 
lytical viewpoint, the superiority of incremental analysis, 
the principal need for discounting costs and benefits, 
and the importance of conducting a sensitivity analysis 
[2]. Besides particular methodological issues such as 
inclusion of indirect costs or choice of discount rate, 
differences between these HEE quality appraisal instru- 
ments refer to the level of aggregation. For instance, 
the International Society for Pharmacoeconomics and 
Outcomes Research (ISPOR) has set up several task 
forces for specific elements of an economic evaluation, 
e.g., the ISPOR Task Force on Good Research Practices 
for Modeling Studies [4]. 

As methods of health economic evaluation mature 
over time, it is also important to appraise and monitor 
the quality of these HEE quality appraisal instruments 
which are used by researchers, journals, institutions and 
governments throughout the world to assess new health 
technologies and allocate resources. To date, tools for 
comparing, measuring, and improving the quality of 
HEE quality appraisal instruments have not been devel- 
oped. Therefore, the objectives of this study are to: es- 
tablish a framework for assessing the quality of HEE 
quality appraisal instruments in order to support and 
improve their quality; apply this quality assessment 
framework to those HEE quality appraisal instruments 
which have been subject to relatively more scrutiny than 
others, in order to test the framework and to demon- 
strate the shortcomings of existing HEE quality appraisal 
instruments. 

Methods 

Before developing a framework for quality assessment of 
HEE quality appraisal instruments, it might be prudent 
to review experiences regarding other applications of 
quality appraisal instruments. The obvious locus for 
such a review is appraisal tools for clinical guidelines. 

To identify potentially relevant clinical guideline ap- 
praisal instruments, the following electronic databases 
were searched from inception to October 2010: PubMed, 
RePEc, and Web of Science. The literature search used 
the following search terms (including synonyms and 



closely related words): "clinical guidelines" combined 
with "appraisal", "instrument", and "quality". Only publi- 
cations in English, French, or German were considered. 
Further publications were retrieved by citation tracking, 
using the "related citations" function in PubMed and 
Web of Science, hand searching the journals "Inter- 
national Journal of Technology Assessment in Health 
Care", "International Journal for Quality in Health Care", 
and "Quality & Safety in Health Care" from 2000 to 
2010, and searching relevant websites. A total of 14 rele- 
vant guideline appraisal instruments were retrieved by 
the search process (see Additional file 1), which are 
described in Table 1. 

However, it was not intended to provide a systematic 
review of appraisal tools for clinical practice guideline 
evaluation. Therefore, the interested reader is referred to 
the reviews by Vlayen et al. [19] and Graham et al. [20], 
which were identified by the literature search on clinical 
guideline appraisal instruments and provide a detailed 
description and comparison of clinical practice guideline 
appraisal instruments. These systematic reviews were 
used to inform the framework development. 

Based on a deductive iterative process, the clinical 
guideline appraisal instruments identified were reviewed, 
consolidated, and adapted. For this purpose, all ques- 
tions/statements included in these instruments were 
listed to exclude double counting. For inclusion in the 
final framework, the questions/statements were required 
to have the following characteristics: 

O Generally and internationally accepted 
O Relevant to the realm of health economic evaluation 
O Distinguishable from other questions/statements 
(i.e., the questions/statements selected for final 
inclusion should overlap as little as possible) 

The questions/statements identified by this meth- 
od were thematically grouped to devise the final quality 
assessment framework for HEE quality appraisal 
instruments. 

In order to test the framework and demonstrate the 
shortcomings of existing HEE quality appraisal instru- 
ments, the framework developed was applied to those 
HEE quality appraisal instruments which have been sub- 
ject to relatively more scrutiny than others. Using a simi- 
lar search process, HEE quality appraisal instruments 
were determined possibly relevant if they provided expli- 
cit criteria against which the quality of economic evalua- 
tions could be appraised. Because of regional, cultural, 
institutional, or political preferences and interests, 
country-specific guidelines were not considered for in- 
clusion. Instruments assessing the transferability of 
health economic evaluations were also beyond the scope 
of this study. 
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Table 2 Framework for quality assessment of HEE quality 
appraisal instruments 



Dimension 


Item 


A. Purpose 


Al. The reasons for developing the 


and scope 


guideline are stated 




A2. The overall objective of the 




guideline is described 




A3. The health economic studies for which 




the guideline was designed are stated 




A4. The target audience of the guideline 




is characterized 




A5. The time frame to which the guideline 




is meant to apply is specified 


B. Stakeholder 


Bl. The guideline development group consists 


involvement 


of individuals from all relevant disciplines 




B2. Conflicts of interest of guideline 




development group members have been 




rprnrrlpH ;^nH ^^HHrpsspH 




B3. The views and preferences or the target 




i^iirlipnrp hs\/p hppn '^nunht 


C. Rigor of 


CI. The members of the guideline development 


development 


group and their affiliations are stated 


process/validity 


C2. The methods used for literature search 




are specified 




C3. The sources of evidence on which the 




guideline is based are described 




C4. The criteria for selecting existing evidence 




are described 




C5. The methods for formulating the items 




are specified 




C6. The methods used to reach consensus 




are specified 




C7. The date for reviewing/updating the 




guideline is stated 




C8. The guideline is valid in terms of 




C8a. Content (internal) validity 




C8b. Criterion (externol) validity 




C8c. Construct validity [convergent and 




discriminant validity) 


D. Reliability/ 


Dl . The process of guideline development 


reproducibility 


is documented 




D2. The guideline has been externally reviewed 




by experts prior to its pub ication 




D3. The guideline has been piloted/pretested 




among the target audience 




ijH. n le guiuciMie is leiiduie iii lciims ui. 




D4a. Inter-rater reliability 




D4b. Test-retest reliability 




D4c. Parallel-forms reliability 




D4d. Internal consistency reliability 


£ Clarity of 


El. The items of the guideline are specific and 


presentation 


clearly worded 




E2. The items of the guideline are clearly 




presented and user friendly 



Table 2 Framework for quality assessment of HEE quality 
appraisal Instruments (Continued) 

E3. The guideline can be used in a 
straightforward manner 

E4. Key items are easily identifiable 

F. Applicability Fl. The guideline provides a standard 

reporting format 

F2. The guideline gives a detailed assessment 
instruction 

F3. The guideline presents items of 
methodological quality and transparency 

F4. The guideline uses a quality score 

F5. The strengths and limitations of the 
guideline are specified 

G. Evaluation G1. The methods for evaluating the guideline 

are described 

G2. The adherence to the guideline by the 
target audience is described 



Results 

Framework for quality assessment of HEE quality 
appraisal instruments 

The framework consists of 36 items organized within 7 
dimensions, each of which captures a specific domain of 
quahty: Dimension A ("purpose and scope", items Al- 
A5), Dimension B {^'stakeholder involvement" , items 
B1-B3), Dimension C {"rigor of development process/ 
validity", items Cl-C8c), Dimension D {"reliability/ 
reproducibility" , items Dl-D4d), Dimension E (^'clarity 
of presentation" , items El-4), Dimension F {^'applicabil- 
ity", items F1-F5), Dimension G ("evaluation", items Gl- 
G2). The quahty assessment framework is presented in 
Table 2. 

Application of the quality assessment framework 

To test the estabUshed framework and to demon- 
strate the shortcomings of existing HEE quahty ap- 
praisal instruments, four well-known, often-cited, and 
widely-used HEE quality appraisal instruments, which 
have been subject to relatively more scrutiny than 
most others, were selected: the Quality of Health 
Economic Studies (QHES) instrument developed by 
Chiou et al. [21], the British Medical Journal (BMJ) 
guidelines for economic submissions established by 
Drummond and Jefferson on behalf of the BMJ Eco- 
nomic Evaluation Working Party [22], the Consensus 
on Health Economic Criteria (CHEC) list devised by 
Evers et al. [23], and the Good Practice Guidelines 
for Decision-Analytic Modeling accomplished by Phi- 
lips et al. [24]. 

The BMJ list, the CHEC list, and the Philips list were 
chosen because they are recommended by the Cochrane 
Handbook for Systematic Reviews of Interventions [25] 
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Table 3 Scope of the four selected HEE quality appraisal 
instruments 





Quality appraisal 
instrument for 
economic evaluation 


Quality appraisal 
instrument for 
systematic reviews 


Trial-based economic 


1,2 


1,2,3 


evaluation 






Model-based economic 


1,2,4 


1,24 


evaluation 







1 QHES instrument [21]. 

2 BMJ guidelines [22]. 

3 CHEC list [23]. 

4 Philips guidelines [24]. 



for critical appraisal of the methodological quality of 
health economic evaluation studies. In the chapter 
related to systematic reviews of economic evaluations, 
the Centre for Reviews and Dissemination's guidance for 
undertaking reviews in health care [26] also refers to the 
BMJ list and the Philips list as instruments to assess the 
quality of economic evaluations. The QHES instrument 
was selected because it is an example of a quality scoring 
system which has been the subject of controversy in the 
literature. 

As can be seen in Table 3, the BMJ list and the QHES 
instrument are broader and more comprehensive than 
the other two quality appraisal instruments: this is be- 
cause the Philips list is solely designed for model-based 
economic evaluations and the CHEC list is only 
intended for undertaking systematic reviews of trial- 
based economic evaluations. In Table 4, the main char- 
acteristics of the HEE quality appraisal instruments are 
provided. 

Pilot review of the quality assessment framework 

The experiences of applying the quality assessment 
framework to the HEE quality appraisal instruments are 
presented in Table 5. 

The quality dimension of "purpose and scope" (dimen- 
sion A) is fulfilled by all HEE quality appraisal instru- 
ments, even though not all items are explicitly 
described. In terms of the quality dimension of "stake- 
holder involvement" (dimension B), only Evers et al. 
(CHEC list, [23]) do not declare whether they have any 
conflicts of interest. Furthermore, at none of the differ- 
ent HEE quality appraisal instrument development 
stages were all the key professionals (e.g., economists, 
clinicians, epidemiologists, and statisticians) involved. 
However, all the quality appraisal instruments used 
methods to ensure that the perspectives of the target 
audience informed the development process (e.g., by 
participation on the development group, or by external 
review of drafts of the appraisal instruments). The main 
differences in the quality of the four HEE quality ap- 
praisal instruments relate to the quality dimension of 



"rigor of development process/validity" (dimension C). 
Especially with the BMJ list, the different stages of the 
development process are not reported. Moreover, as 
methodology advances, the date for updating the ap- 
praisal instrument should be stated. Only Philips et al. 
[24] refer to the need for periodic updates, but when 
these should take place remains unclear. The lack of for- 
mal validity is one of the main limitations of all four 
quality appraisal instruments. Only the QHES instru- 
ment was formally validated in terms of construct valid- 
ity [21]. Other limitations refer to the quality dimension 
of "reliability /reproducibility" (dimension D). As men- 
tioned before, the development process of the BMJ list is 
not documented. All four HEE quality appraisal instru- 
ments were published in peer-reviewed journals and, 
hence, external reviewers were involved in appraisal in- 
strument development during the generic review 
process. However, in order to develop the BMJ list and 
the Philips list, additional external experts were con- 
vened to discuss drafts of these quality appraisal instru- 
ments. In addition, provided that external reviewers 
should not have been involved in developing the ap- 
praisal instrument, the QHES instrument and the CHEC 
list have not been externally reviewed before their publi- 
cation (certainly, except for the review process). For the 
appraisal instrument to be effective with regard to reli- 
ability and reproducibility, it also needs to be piloted/ 
pretested among the target audience. Only the QHES in- 
strument was pretested among 60 experts in the field of 
health economics, who evaluated the methodological 
quality of three health economic analyses, first on a vis- 
ual analog scale, and then using the grading system. In 
respect of the quality dimension of "clarity of presenta- 
tion" (dimension E), the appraisal instruments estab- 
lished by Philips et al. [24] and the grading system 
developed by Chiou et al. (QHES instrument, [21]) do 
not provide specific and unambiguous items in those 
cases where more than one question refers to the same 
criterion, resulting in ambivalent assessments. Addition- 
ally, on account of the rather technical nature of the 
questions provided by Philips et al. [24], these are only 
suitable for specialist readers with expertise in the field 
of decision-analytic modelling and with knowledge of 
the disease area. Philips et al. [24] state that without that 
knowledge, it is a complex matter to decide whether all 
structural assumptions are justified, or whether all feas- 
ible and practical options have been evaluated. Further, 
because of the problem with the interpretation of the 
term "justified" or "appropriate", it might be difficult to 
use these four quality appraisal instruments in a straight- 
forward manner. Some items/questions are highly 
dependent on the judgment of the respective user and 
thus have an unavoidable element of subjectivity. For ex- 
ample, based on a comparison of three instruments for 
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Table 4 Main characteristics of the four selected HEE quality appraisal instruments 



QHES instrument [21] 



BIVIJ guidelines [22] 



CHEC list [23] 



Philips guidelines [24] 



Author; year; journal 



Purpose 



Development process 



Time frame 
Target audience 



Preferred analytical 
technique 



Standard reporting 
format included 



Chiou et al. 2003; Medical 
Care 



Affiliation of authors 

Published in a 
peer-reviewed journal 

Number of references 35 



Academia and industry 
Yes 



Drummond, Jefferson; 
1996; British Medical 
Journal (BMJ) 

Academia 

Yes 



Evers et al. 2005; International Philips et al. 2006; 
Journal of Technology Pharmacoeconomics 
Assessment in Health Care 



To provide a grading system To improve the 

for assessing the quality of transparency of reporting 

health economic evaluations 



Selection of criteria from 19 Not specified 

existing guidelines; Use of a 

conjoint analysis survey of 

120 international experts to 

estimate weights for each 

criterion 

Before, during and after Before, during, and 
peer review after peer review 

Producers and consumers of Producers and consumers 
economic evaluations of economic evaluations 



Full economic evaluations: 
cost-minimization-, cost- 
effectiveness-, cost-utility-, 
cost-benefit-analysis 

16 questions which should 
be asked when appraising 
the quality of health 
economic evaluations 



Full economic evaluations: 
cost-minimization- cost- 
effectiveness-, cost-utility-, 
cost-benefit-analysis 



Academia 
Yes 

30 

To develop a generally 
accepted criteria list for 
assessing the methodology 
of economic evaluation 
studies in systematic 
reviews 

Selection of items from 15 
existing guidelines; Use of a 
Delphi panel consisting of 
23 international experts to 
generate the final criteria list 

After peer review 

Consumers intending to 
conduct a systematic review 
of trial-based economic 
evaluations 

Full economic evaluations 
based on clinical trials: cohort based on decision-analytic 
studies, case-control studies, models 
randomized controlled 
clinical trials 



Number of 1 6 
questions/criteria 

Operationalizatlon of Yes/No 
the questions/criteria 

Use of a quality score Yes 

Assessment instruction No 



Academia 
Yes 

22 

To identify, review, and consolidate 
currently available guidelines in 
order to establish a synthesized 
and consistent quality assessment 
framework for decision analytic 
models 

Selection and formulation of items 
by reviewing and consolidating 15 
existing guidelines for good 
practice in decision-analytic 
modeling in HTA 

Before, during, and after 
peer review 

Producers and consumers of 
model-based economic 
evaluations 

Full economic evaluations 



P/7or fesf of the 
guideline 



Yes: Ofman et al. [31 



Ten sections under the three 
headings of study design, 
data collection, and analysis 
and interpretation of results: 
study question, selection of 
alternatives, form of 
evaluation, effectiveness data, 
benefit measurement and 
valuation, costing, modeling, 
adjustments for timing of 
costs and benefits, allowance 
for uncertainty, and 
presentation of results 

35 

Yes/No/Not clear/Not 
appropriate 

No 

Yes 

Not specified 



19 questions which should 
be asked when appraising 
the quality of health 
economic evaluations in 
systematic reviews 



Yes/No 



No 



15 sections under the three key 
themes of structure, data, and 
consistency: statement of decision 
problem/objective, statement of 
scope/ perspective, rationale for 
structure, structural assumptions, 
strategies/comparators, model 
type, time horizon, disease 
states/pathways, cycle length, 
data identification, pre-model 
data analysis, data incorporation, 
assessment of uncertainty, internal 
consistency, and external 
consistency 

61 



Yes/No/Unclear/Not applicable 



Yes: http://www.beoz. 
unimaas.nl/chec/ 

Yes, but no details given 



No 

Yes 



Yes 



appraising the quality of health economic evaluations 
(BMJ list, CHEC list, and QHES instrument) Gerkens 
et al. [27] found that the reviewer has a greater influence 
on the results of the quality assessment than the 



appraisal instrument itself Another problem within this 
dimension concerns the operationalizatlon of the ques- 
tions. Chiou et al. (QHES instrument, [21]) and Evers 
et al. (CHEC list, [23]) provide questions in a yes/no 



Langer BMC Health Services Research 2012, 12:253 
http://www.biomedcentral.com/1472-6963/12/253 



Page 8 of 1 1 



Table 5 Application of the quality assessment framework to the four selected HEE quality appraisal instruments 


Dimension 


Item 


QHES instrument [21] 


BIVIJ guidelines [22] 


CHEC list [23] 


Philips guidelines [24] 


A. Purpose and scope 


Al 


YES 


YES 


YES 


YES 




A2 


YES 


YES 


YES 


YES 




A3 


YES 


YES 


YES 


YES 




A4 


YES 


YES 


YES 


YES 




A5 


YES 


YES 


YES 


YES 


B. Stakeholder Involvement 


B1 


NO 


NO 


NO 


NO 




B2 


YES 


YES 


NO 


YES 




B3 


YES 


YES 


YES 


YES 


C. Rigor of development 


CI 


YES 


PARTIALLY 


YES 


YES 


process/validity 


C2 


YES 


NO 


YES 


YES 




C3 


YES 


NO 


YES 


YES 




C4 


YES 


NO 


NO 


YES 




C5 


YES 


NO 


YES 


YES 




C6 


YES 


NO 


YES 


YES 




C7 


NO 


NO 


NO 


NO 




C8 












C8a 


NO 


NO 


NO 


NO 




C8b 


NO 


NO 


NO 


NO 




C8c 


YES 


NO 


NO 


NO 


D. Rellabllity/reproduclbllity 


D1 


YES 


NO 


YES 


YES 




D2 


NO 


YES 


NO 


YES 




D3 


YES 


NO 


NO 


NO 




D4 












D4a 


NO 


NO 


NO 


NO 




D4b 


NO 


NO 


NO 


NO 




D4c 


NO 


NO 


NO 


NO 




D4d 


NO 


NO 


NO 


NO 


E. Clarity of presentation 


El 


NO 


YES 


YES 


NO 




E2 


YES 


YES 


YES 


NO 




E3 


NO 


NO 


NO 


NO 




E4 


YES 


NO 


NO 


NO 


F. Applicability 


F1 


YES 


YES 


YES 


YES 




F2 


NO 


YES 


YES 


YES 




F3 


YES 


YES 


YES 


YES 




F4 


YES 


NO 


NO 


NO 




F5 


YES 


NO 


YES 


YES 


G. Evaluation 


G1 


NO 


YES 


YES 


YES 




G2 


NO 


NO 


NO 


NO 



format, but in specific circumstances some questions 
may not be applicable to the study context. Because of 
the weighting of the criteria, only the key items of the 
QHES instrument can be easily identified. Regarding the 
quality dimension of "applicability" (dimension F), all 
appraisal instruments provide a standard reporting 



format and present items of methodological quality and 
transparency. All but the QHES instrument give detailed 
assessment instructions (to a greater or lesser extent) 
and all appraisal instruments except for the BMJ list spe- 
cify their strengths and limitations. The QHES instru- 
ment is the only quality appraisal instrument to use a 
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quality score. Concerning the quality dimension of 
"evaluation" (dimension G), all but the QHES instru- 
ment describe the evaluation methods. However, none of 
the instruments describes the extent of adherence by the 
target audience. 

Discussion 

At present, there is no common instrument for measur- 
ing and improving the quality of HEE quality appraisal 
instruments. A quality assessment framework for HEE 
quality appraisal instruments was developed to support 
and improve their quality. It permits not only the assess- 
ment of their quality but also the recognition of the 
most urgent adjustments needed to improve their qual- 
ity. Applying the quality assessment framework to four 
existing HEE quality appraisal instruments, it was found 
that these quality appraisal instruments are of variable 
quality. 

Moreover, the HEE quality appraisal instruments have 
other limitations. The CHEC list established by Evers 
et al. [23] consists of a minimum set of items and is 
intended only for full economic evaluations based on 
clinical trials. In order to appraise the overall methodo- 
logical quality of trial-based economic evaluations, the 
authors point out that their list should be used in com- 
bination with existing instruments for assessing the 
quality of clinical trials [23]. In systematic reviews in- 
cluding trial-based and model-based economic evalua- 
tions, the CHEC list should be combined with issues 
relevant to modelling studies such as structural assump- 
tions. Therefore, in a systematic review [28], the ap- 
praisal instruments developed by Philips et al. [29] were 
used as a complement to the CHEC list. Another limita- 
tion concerns the items included in the CHEC list. As 
these items should deliver insight into the quality of eco- 
nomic evaluation studies, most of them are rather sub- 
jective, which is a challenge for the inter-rater reliability 
[23]. However, formularies and HTAs would in fact need 
some flexibility to make their own best decisions for 
their patients. The subjective judgment generally inher- 
ent in quality assessment is a particularly fundamental 
problem for the Philips guidelines [24]. Additionally, be- 
cause of the combination of transparency and quality 
aspects in the same question, the Philips guidelines pro- 
duce ambivalent quality assessments. By contrast, other 
research groups provide quality appraisal instruments 
that differentiate between these two issues [30]. Further- 
more, because of the scope of the Philips list, it should 
be used in conjunction with more general quality assess- 
ment instruments for health economic evaluation (e.g., the 
BMJ list) [24]. The quality appraisal instrument devised 
by Philips et al. [24] include dimensions of methodo- 
logical quality corresponding to rationales for structure, 
structural assumptions, disease states/pathways, cycle 



length, and internal consistency. This is due to the fact 
that this HEE quality appraisal instrument is specific for 
decision-analytic models, and, thus, has a more tech- 
nical character than the other three HEE quality ap- 
praisal instruments. Therefore, the Philips list does not 
highlight the importance of discounting, the superiority 
of incremental analysis, and the measurement and 
valuation of costs. However, in their version published 
in 2004, Philips et al. [29] point out that "costing and 
discounting methods should accord with standard 
guidelines for economic evaluation". Due to limitations 
in reporting, the quality of the BMJ list in particular was 
difficult to assess. In contrast to the other three quality 
appraisal instruments, the QHES instrument provides a 
grading system, but the advantage of scoring methods is 
questionable. 

To date, relatively little empirical research has been 
undertaken in order to investigate the influence of deci- 
sions to include only economic evaluations of high qual- 
ity on the results of a critical assessment of health 
economic evaluations. However, such lessons can be 
obtained from the experiences made with quality scores 
for clinical studies. Using 25 different quality assessment 
scales to identify high-quality clinical trials, Jiini et al. 
[32] show that the conclusions of meta-analytic studies 
of randomized clinical trials might be affected by the 
type of quality assessment scale used. They consider the 
use of grading systems to be problematic, and thus they 
recommend that relevant methodological issues should 
be appraised individually. In a review, Moher et al. [33] 
use the same 25 scales to show differences in scale de- 
velopment. As these differences can result in important 
differences in quality assessment, they recommend that 
meta-analyses of randomized clinical trials should be 
undertaken with and without assessing quality. Based on 
these experiences, it is not recommended to select eco- 
nomic evaluations on the basis of their quality scores, as 
proposed by Chiou et al. (QHES instrument, [21]. In 
general, a corresponding NHS EED structured abstract 
[34] consisting of "subject of study", "key elements of 
study", "details about clinical evidence", "economic ana- 
lysis", "results", "critical commentary", "implications", 
and "other publications of related interest" might en- 
hance quality assessment of all types of full health eco- 
nomic evaluation informed by HEE quality appraisal 
instruments, because these abstracts provide critical ap- 
praisal of methodological quality on the basis of the 
same quality dimensions as included in the quality ap- 
praisal instruments [25]. In their study, Thurston et al. 
[35] found that decision-makers in health care need an 
initial screen of economic evaluation results provided by 
a critical descriptive summary or a score, plus a short 
abstract to gather more information on the quality and 
relevance of economic evaluation results. However, how 
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to condense information provided by critical appraisal of 
methodological quality is an unresolved issue which 
requires further research. 

Conclusions 

The framework described in this study should be 
regarded as a starting point for assessing the quality of 
HEE quality appraisal instruments. This framework can 
be used by quality appraisal instrument producers to 
support and improve the quality and acceptance of exist- 
ing and future HEE quality appraisal instruments. By ap- 
plying this framework, users of HEE quality appraisal 
instruments can become aware of methodological defi- 
ciencies inherent in existing quality appraisal instruments, 
as illustrated by the pilot test. As the development of HEE 
quality appraisal instruments is a dynamic and interactive 
process, and as methodology advances, a continual update 
of existing quality appraisal instruments is needed. 

Additional file 



Additional file 1: Flow chart for study selection for clinical practice 
guideline evaluation. Flow chart showing the records identified 
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